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INFORMATION INPUT APPARATUS, INFORMATION INPUT METHOD, 
AND RECORDING MEDIUM 



This application is based on Japanese Patent 
Application No. 10-66382, filed March 17, 1998, the 
contents of which are incorporated herein by reference. 

The present invention relates to an information 
input apparatus and method for inputting information in 
a three-dimensional space, and to a recording medium. 

As an input device to a computer, a mouse is 
prevalently used. However, the mouse is used to merely 
attain roles of a two-dimensional pointing device such 
as movement of the cursor, selection of a menu, and the 
like. Since information the mouse can process is 
two-dimensional information, the mouse can hardly 
select, e.g., an object with a depth in a 
three-dimensional space. On the other hand, when the 
mouse is used to animate a character upon creating an 
animation, it cannot easily naturally animate the 
character. In order to compensate for such difficul- 
ties in pointing in a three-dimensional space, 
three-dimensional pointing devices have been developed. 
For example, a three-dimensional pointing device shown 
in FIG. 6 allows six ways of operations, i.e., pushing 
a central round portion forward, pressing the center of 
that portion, pressing the rear end of that portion. 
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lifting the entire portion upward, turning the entire 
portion clockwise, and turning the entire portion 
counterclockwise, and has six degrees of freedom. By 
assigning these six degrees of freedom to various 
instructions, the position (x, y, 2) and directions (x-, 
y-, and z-axes) of a cursor in a three-dimensional 
space can be controlled, or the view point position (x, 
y, z) and directions (x-, y- , and z-axes) with respect 
to the three-dimensional space can be controlled. 

However, when this device is operated actually, 
the cursor or view point can'not be desirably controlled. 
For example, when the operator wants to turn the round 
portion clockwise or counterclockwise, he or she may 
press its forward or rear end, and the cursor or view 
point may move in an unexpected direction. 

In place of such three-dimensional pointing device, 
devices that can input instructions using hand or body 
actions have been developed. Such devices are called, 
e.g., a data glove, data suit, cyber glove, and the 
like. For example, the data glove is a glove-like 
device, and optical fibers run on its surface. Each 
optical fiber runs to a joint of each finger, and upon 
bending the finger, the transmission state of light 
changes. By measuring the transmission state of light, 
the bent level of the joint of each finger can be 
detected. The position of the hand itself in the 
three-dimensional space is measured by a magnetic 



sensor attached to the back of the hand. If an action 
is assigned to a given instruction (e.g., if the index 
finger is pointed up, a forward movement instruction is 
issued) , the operator can walk in the three-dimensional 
space by variously changing the view point using the 
data glove (walkthrough) . 

However, some problems must be solved. Such 
device is expensive, and can hardly be used for home 
use. Since the angle of the finger joint is measured, 
even when, for example, stretching only the index 
finger and bending other fingers is defined as a 
forward movement instruction, stretching a finger 
includes various states. That is, since the second 
joint of the index finger rarely makes 180°, it is 
different to recognize the stretched state except for 
such 180° state of the index finger, unless a given 
margin is assured. Since the operator must wear the 
data glove, his or her natural movement is disturbed. 
Every time the operator wears the data glove, he or she 
must calibrate the transmission state of light in 
correspondence with the stretched and bent finger 
states, resulting in troublesome operations. Since 
optical fibers are used, failures such as disconnection 
of fibers may take place after continuous use of the 
data glove, and the data glove has a durability as low 
as an expendable. Despite the fact the data glove is 
such expensive, troublesome device, if the glove size 



does not just fit with the operator's hand, the input 
value may deviate from the calibrated value during use 
due to slippage of the glove, and delicate hand actions 
can hardly be recognized. Owing to various problems 
described above, the data glove has not so prevailed 
contrary to initial expectation although it served as a 
trigger device of the VR (virtual reality) technology. 
For this reason, the data glove is still expensive, and 
has many problems in terms of its use. 

By contrast, some studies have been made to input 
hand and body actions without wearing any special 
devices such as a data glove. For example, a method of 
recognizing hand shape by analyzing a moving image such 
as a video image has been studied. 

However, with such method, it is very hard to 
extract an objective image portion (e.g., in case of 
hand action recognition, a hand image alone) from the 
background image. For example, assume that an 
objective image is extracted using colors. Since the 
hand has skin color, only a skin color portion may be 
extracted. However, if a beige clothing article or 
wall is present as a background, it is hard to 
recognize skin color. Even when beige is distinguished 
from skin color by adjustment, if illumination changes, 
the color tone also changes. Hence, it is difficult to 
steadily extract a skin color portion. 

In order to avoid such problems, a method that 



facilitates extraction by imposing a constraint on the 
background image, e.g., by placing a blue mat on the 
background may be used. Alternatively, a method that 
colors finger tips to easily extract them from the 
background or makes the operator wear color rings may 
be used. However, such constraints are not practical; 
they are used for experimental purposes but are not put 
into practical applications. 

The above-mentioned video image recognition such 
as extraction and the like requires a very large 
computation amount. For this reason, existing personal 
computers cannot process all video images (as large as 
30 images per sec) in real time. Hence, it is hard to 
attain motion capture by video image processing in real 
time . 

A device called a range finder for inputting a 
distant image is known. The typical principle of the 
range finder is to irradiate an object with spot light 
or slit light and obtain a distant image based on the 
position where the light reflected by the object is 
received by the principle of triangulation . The range 
finder mechanically scans spot light or slit light to 
obtain two-dimensional distance information. This 
device can generate a distant image with very high 
precision, but requires a large-scale arrangement, 
resulting in high cost. Also, a long input time is 
required, and it is difficult for this device to 



process information in real time. 

A device for detecting a color marker or 
light-emitting unit attached to a hand or body portion 
from an image, and capturing the shape, motion, and the 
like of the hand or body portion may be used, and has 
already been put into some applications. However, the 
device has a serious demerit of user's inconvenience, 
since the user must wear the device upon every 
operation, and the application range is limited very 
much. As in the example of the data glove, when the 
user wears the device on hi^ or her movable portion 
such as a hand, the durability problem, is often posed. 

The problems in a conventional camera technique 
will be explained below in addition to the aforemen- 
tioned input devices . With the conventional camera 
technique, in order to synthesize (chromakey) a 
character with a background, a character image must be 
photographed in front of a blue back to facilitate 
character extraction. For this reason, the photograph- 
ing place is limited to, e.g., a studio that can 
photograph an image in front of a blue back. 
Alternatively, in order to extract a character from an 
image photographed in a non-blue back state, the 
character extraction range must be manually edited in 
units of frames, resulting in very cumbersome 
operations , 

Similarly, in order to generate a character in a 



three-dimensional space, a three-dimensional model is 
created in advance, and a photograph of the character 
is pasted to the model (texture mapping) . However, 
creation of a three-dimensional model and texture 
mapping are tedious operations and are rarely used 
other than applications such as movie production that 
justifies extravagant cost needed. 

In order to solve these problems, for example, a 
technique disclosed in USSN 08/953,667 is known. This 
technique acquires a distant image by extracting a 
reflected light image. However, this technique cannot 
obtain hue information of an object since it extracts 
the reflected light image. For this reason, two 
different types of cameras, i.e., a conventional 
imaging camera and a camera for extracting a reflected 
light image, are required. 

BRIEF SUMMARY OF THE INVENTION 
It is an object of the present invention to 
provide an information input apparatus and method that 
can acquire a reflected light image using a versatile 
image sensor, and a recording medium. 

It is another object of the present invention to 
provide an information input apparatus and method which 
can attain high-level image processing such as 
extraction of an object image alone from the background 
in a normal image, and the like, and a recording medium. 
In order to achieve the above objects, according 



to the first aspect of the present invention, an 
information input apparatus comprises : a light emitter 
for irradiating an object with light; an area image 
sensor for outputting a difference between charges 
received by light-receiving cells arranged in an array 
pattern from a reflected light of the object caused by 
the light emitter irradiating the object with light; a 
timing signal generator for generating a timing signal 
comprised of a pulse signal or a modulation signal for 
controlling an intensity of light of the light emitter; 
a control signal generator for generating a control 
signal for individually controlling light-receiving 
timings of the light-receiving cells of the area image 
sensor on the basis of the timing signal from the 
timing signal generator; and image processing section 
for extracting a reflected light image of the object 
from the difference outputted from the area image 
sensor . 

According to the second aspect of the present 
invention, an information input apparatus comprises: a 
timing signal generator for generating a timing signal 
comprised of a pulse signal or a modulation signal; a 
light emitter for emitting light, an intensity of which 
changes on the basis of the timing signal from the 
timing signal generator; first light-receiving section 
for receiving light emitted by the light emitter and 
reflected by an object in synchronism with the timing 
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signal from the timing signal generator; and second 
light-receiving section for receiving light other than 
the light emitted by the light emitter and reflected by 
the object. 

5 According to the third aspect of the present 

invention, an information input method comprises the 
steps of: generating a pulse signal or a modulation 
signal; generating, on the basis of the pulse or 
modulation signal, a control signal for separately 

10 controlling light-receiving timings of light-receiving 

cells of an area image sensor for obtaining a 
difference between charges received by light-receiving 
cells which are arranged in an array pattern; emitting 
light, an intensity of which changes on the basis of 

15 the generated control signal; and detecting a light 

image reflected by an object of the emitted light. 

According to the fourth aspect of the present 
invention, an information input method comprises the 
steps of: generating a pulse signal or modulation 

20 signal; emitting light, an intensity of which changes 

on the basis of the pulse or modulation signal; and 
receiving light reflected by an object of the emitted 
light and light other than the reflected light in 
synchronism with the pulse or modulation signal. 

25 According to the fifth aspect of the present 

invention, an article of manufacture comprises: a 
computer usable medium having computer readable program 



code means embodied therein for causing an area image 
sensor for obtaining a difference between charges 
received by light-receiving cells which are arranged in 
an array pattern to be controlled, the computer 
readable program code means in the article of 
manufacture comprising: computer readable program code 
means for causing a computer to generate a pulse signal 
or a modulation signal; computer readable program code 
means for causing a computer to generate a control 
signal for separately controlling light-receiving 
timings of the light-receiving cells of the area image 
sensor on the basis of the pulse or modulation signal; 
computer readable program code means for causing a 
computer to cause a light emitter to emit light, an 
intensity of which changes on the basis of the 
generated pulse signal or modulation signal; and 
computer readable program code means for causing a 
computer to extract a light image reflected by an 
object of the emitted light from the difference 
outputted from the area image sensor. 

According to the present invention, since a 
reflected light image can be acquired using a versatile 
image sensor, i.e., the versatile image sensor can be 
used, a cost reduction of the apparatus can be realized. 

According to the present invention, high-level 
image processing such as extraction of an object image 
alone from the background in a normal image, and the 
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like can be easily implemented. 

Furthermore, according to the present invention, a 
reflected image and an image based on other light 
components can be simultaneously obtained, 
5 Additional objects and advantages of the invention 

will be set forth in the description which follows, and 
in part will be obvious from the description, or may be 
learned by practice of the invention. The objects and 
advantages of the invention may be realized and 
10 obtained by means of the instrumentalities and combina- 

tions particularly pointed Out hereinafter. 

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING 
The accompanying drawings, which are incorporated 
in and constitute a part of the specification, illus- 
15 trate presently preferred embodiments of the invention, 

and together with the general description given above 
and the detailed description of the preferred embodi- 
ments given below, serve to explain the principles of 
the invention - 

20 FIG. 1 is a perspective view showing an example of 

a conventional three-dimensional data input device; 

FIG. 2 is a block diagram showing an example of 

the arrangement for acquiring a reflected light image 

using an image sensor; 
25 FIG. 3 is a block diagram showing an example of 

the arrangement for acquiring a reflected light image 

using an image sensor in more detail; 
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FIG. 4 is a flow chart showing the operation for 
acquiring a reflected light image using an image 
sensor; 

FIG. 5 shows an example of the acquired reflected 
light image; 

FIG. 6 is a block diagram showing an example of 
the arrangement of an apparatus that can simultaneously 
acquire an image based on object reflected light and an 
image based on other light components; 

FIG. 7 is a block diagram showing an example of 
the arrangement of an apparatus that can simultaneously 
acquire an image based on object reflected light and an 
image based on other light components; 

FIG. 8 is a block diagram showing an example of 
the arrangement of an apparatus that can simultaneously 
acquire an image based on object reflected light and an 
image based on other light components; 

FIG. 9 shows an example of the filter arrangement 
on an image sensor of the apparatus that can 
simultaneously acquire an image based on object 
reflected light and an image based on other light 
components ; 

FIG. 10 shows an example of the filter arrangement 
on an image sensor of the apparatus that can 
simultaneously acquire an image based on object 
reflected light and an image based on other light 
components ; 



FIG. 11 is a block diagram showing an example of 
the arrangement of an apparatus that can simultaneously 
acquire an image based on object reflected light and an 
image based on other light components; 

FIG. 12 is a block diagram showing an example of 
the arrangement of an apparatus that can simultaneously 
acquire an image based on object reflected light and an 
image based on other light components; 

FIG. 13 shows an example of the filter arrangement 
on an image sensor of the apparatus that can 
simultaneously acquire an iiUage based on object 
reflected light and an image based on other light 
components; and 

FIG. 14 shows an example of the filter arrangement 
on an image sensor of the apparatus that can 
simultaneously acquire an image based on object 
reflected light and an image based on other light 
components . 

DETAILED DESCRIPTION OF THE INVENTION 
The preferred embodiments of the present invention 
will be explained hereinafter with the aid of the 
accompanying drawings . 

FIG. 2 shows an example of the arrangement of an 
apparatus for acquiring a reflected light image using 
an image sensor. A light emitting controller 8 
controls a light source 7 to emit light in a 
predetermined light emitting pattern on the basis of a 
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timing signal comprised of a pulse signal or modulation 
signal generated by a system controller 11. More 
specifically, the light source emits one-shot pulse 
light per frame. The wavelength of light is not 
5 particularly limited, but a light source that emits 

light such as near infrared light which is contained in 
external light (illumination light, sunlight), has low 
power, and does not glare, is preferably used. 

The light source 7 preferably comprises, e.g., an 

10 LED which stably operates for a long period of time, 

and responds at high speed. ' A light-receiving sensor 1 
operates to sense at least twice in synchronism with 
the light emitted by the light source 7. The sensor 1 
operates simultaneously with light emission to receive 

15 light containing object reflected light 5 of light 14 

emitted by the light source. The sensor 1 also 
operates when the light source ceases to emit any light, 
so as to receive light that does not contain any object 
reflected light of the light emitted by the light 

20 source. This sensor can output the difference between 

these two imaging results. That is, by obtaining this 

o 

difference, the object reflected light alone of the 
light emitted by the light source is output. An 
intensity of the light emitted by the light source is 
25 controlled based on the timing signal. 

A control signal generator 10 generates control 
signals for controlling light reception of individual 
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light-receiving cells of the light-receiving sensor 1 
on the basis of the pulse or modulation signal 
generated by the system controller 11^ and supplies 
these signals to the light-receiving sensor 1. 

A filter 2 that intercepts unwanted light is 
inserted between a lens 3 and the sensor 1 • For 
example, an optical bandpass filter that passes only 
the light source wavelength is inserted. For this 
reason, most of unwanted light components 4 such as 
illumination light do not become incident on the sensor 
However, since external light such as illumination 
light or the like normally has light components haying 
the same wavelength as the light source wavelength, the 
difference is output. 

The sensor output is converted into digital data 
by an A/D converter 12 via an analog signal processor 9 
A reflected light image processor 13 performs various 
kinds of processing using the digital image data. Upon 
interpreting the operator's hand action, the processor 
13 extracts a two- or three-dimensional hand shape, and 
estimates the hand action. A detailed description of 
the processing in this reflected light image processor 
13 will be omitted in this embodiment. 

FIG. 3 shows an image sensor in the 
light-receiving sensor unit shown in FIG. 2 in more 
detail. The image sensor shown in FIG. 2 can designate 
pixel position by a vertical selector 15 and horizontal 
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selector 17 and can extract an output stored in the 
designated cell. 

A difference circuit 18 can temporarily store a 
charge amount and can output the difference between the 
stored charge amount and the next input charge amount. 
In this image sensor, imaging operations of even and 
odd lines can be separately controlled. The even lines 
are used as cells that receive light when the light 
source emits light, and the odd lines are used as cells 
that receive light when the light source ceases to emit 
light. More specifically, the cells in the even lines 
sense when the light source emits light, and the cells 
in the odd lines sense when the light source ceases to 
emit light. The former cells will be referred to as 
emission-timing storage cells 20, and the latter cells 
will be referred to as non-emission-timing storage 
cells 19 hereinafter. 

For example, the difference circuit 18 operates as 
follows. The difference circuit 18 reads out contents 
of non-emission-timing storage cells in the first 
line, and stores them for one line. Then, the 
difference circuit 18 reads out the contents of 
non-emission-timing storage cells in the second line, 
and outputs the difference between the currently 
readout amount and the stored received-light amount for 
the first line. The subsequent processing is the same 
as that in FIG- 2 . 
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FIG. 4 is a flow chart showing the processing in 
the arrangement shown in FIG. 3. When the LED emits 
light (step SlOl), the even lines receive light (step 
S102); when the LED ceases to emit light (step S103), 
the odd lines receive light (step S104). In this way, 
as shown in FIG. 3, charges received while light is 
emitted are stored in the even lines, and those 
received while light is not emitted are stored in the 
odd lines. These charges are read out via the 
difference circuit 18. 

Initially, the received-light level of one odd 
line, i.e., at a non-emission timing is stored in the 
difference circuit (step S106). Then, the 
received-light level of one even line, i.e., at an 
emission timing are read out to the difference 
circuit 18 (step S107), and only the difference between 
the levels at the emission and non-emission timings is 
output (step S108). The difference is input to the 
analog signal processor 9 (step S109). 

Such operation continues until the contents of all 
the lines are transferred in the vertical direction. 
Upon completion of transfer, operation for emitting and 
receiving light is repeated. 

With the above-mentioned operation, for example, a 
reflected light image (a three-dimensional image in 
practice since the intensity of reflected light is 
inversely proportional to the square of distance) shown 



in FIG. 5 is acquired. 

In FIG. 3/ the apparatus for acquiring an object 
reflected light image of a light source using an image 
sensor has been described. However, there are 
requirements for an apparatus that can simultaneously 
obtain a reflected light image from which the object 
shape can be easily acquired, and an image taken by a 
normal camera (to be referred to as a visible light 
image hereinafter) . If such apparatus is realized, a 
human body or face alone can be easily extracted from 
the visible light image. 

FIG. 6 shows an example of the arrangement of an 
apparatus which can simultaneously obtain a reflected 
light image and visible light image. In this 
embodiment, two different types of image sensors are 
prepared in correspondence with the reflected light 
image and visible light image. 

A light-receiving sensor 1 for the reflected light 
image has the arrangement described above with the aid 
of FIG. 3. An optical bandpass filter 2 that passes 
only the wavelength light of a light source is placed 
in front of the image sensor 1. This filter passes 
only object reflected light 5 of the light source, and 
reflects other light components such as illumination 
light and its object reflected light 4. 

On the other hand, a sensor 22 for the visible 
light image can use a CCD image sensor, CMOS image 



sensor, and the like, which are used in a normal 
imaging camera. An optical filter 23 that reflects 
object reflected light from the light source is 
normally placed in front of the sensor for the visible 
light image. However, when the object reflected light 
of the light source does not influence largely, e.g., 
when an infrared light cut filter used in an imaging 
camera suffices or when the sensor for the visible 
light image has low sensitivity to the light source due 
to its spectral sensitivity characteristics, this 
optical filter may be omitted. 

When these two sensors simultaneously operate^ 
higher-level information such as an extracted object 
can be obtained. Also, this apparatus can be used for 
simply obtaining a reflected light image or visible 
light image alone. 

In this arrangement, since the sensor for the 
reflected light image is placed in the vicinity of the 
sensor for the visible light image, images nearly free 
from position errors can be obtained. However, 
strictly speaking, slight position errors are present, 
and they become conspicuous if the object is close to 
the apparatus . 

FIG. 7 shows an example of the arrangement that 
can prevent such position errors. In this arrangement, 
an optical system is shared by the reflected light 
image and visible light image. Light that has passed 



through the optical system is split into object 
reflected light 29 of the light source and other light 
components 30 by, e.g., a dichroic mirror 28. The 
dichroic mirror passes light in a specific wavelength 
range, and reflects other light components. The 
arrangement shown in FIG. 7 uses a mirror that passes 
the wavelength light of the object reflected light, and 
reflects other light components. With this arrangement, 
two images having optical axes which accurately match, 
i.e., a reflected light image and visible light image, 
can be obtained. 

In this example, a dichroic mirror is used. 
However, the present invention is not limited to such 
specific device but can use any other devices which can 
split light into the object reflected light of the 
light source and other light components, and can input 
them to two image sensors placed at positions where 
they do not overlap each other. 

In FIGS . 6 and 7 , the two sensors respectively for 
the reflected light image and visible light image are 
used- If such processing can be done using a single 
sensor, further cost reduction can be attained. FIG. 8 
shows one example of such arrangement. A slidable 
optical filter 31 is inserted between a lens 32 and 
sensor 33. 

This filter has a bandpass filter that passes only 
the wavelength light of the light source on its half 



portion, and has an optical filter that intercepts this 
wavelength light and passes visible light on the other 
half portion. By sliding this filter,, light that 
reaches the sensor can be limited to only the object 
reflected light of the light source or visible light. 
In this case, a single image sensor must sense both a 
reflected light image and visible light image. Such 
sensor will be described below. 

FIG. 9 shows a color filter matrix normally used 
in a CMOS image sensor. As shown in FIG. 9, this 
matrix includes "R" cells that pass red components, "G" 
cells that pass green components, and "B" cells that 
pass blue components. When a visible light image is to 
be sensed, the outputs from these cells are directly 
output. During this period, the difference circuit 
ceases to operate or is used only for removing fixed 
pattern noise. The fixed pattern noise is the one that 
has fixed values in units of cells. 

In order to obtain a reflected light image, the 
sensor operation is switched. For example, only "G" 
cells are used. In this case, a Gl cell stores a 
non-emission-timing charge, a G2 cell stores an 
emission-timing charge, and their difference can be 
output using the difference circuit. For this purpose, 
imaging control of G cells must be done using two 
systems to separately sense images . 

For example, a group of cells Gl, G3, G5 , and G7, 
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and a group of cells G2 , G4 , G6 , and G8 can separately 
sense images. In this case, the visible light image 
and reflected light image have different image 
resolutions. In case of the visible light image, since 
5 four pixels (two G pixels and one each R and B pixels) 

form one color pixel, the matrix shown in FIG. 9 
includes 4x4 pixels. When the differences between Gl 
and G2 and between G3 and G4 are calculated, and are 
added to each other, i.e., when Gl through G4 form one 

10 pixel of the reflected light image, the resolution is 

halved in the vertical direction, and remains the same 
in the horizontal direction. When Gl through G8 form 
one pixel of the reflected light image, the resolution 
is halved in both the vertical and horizontal 

15 directions (2 x 2 in FIG. 9). 

In the above example, the reflected light image is 
obtained using "G" cells alone. However, when the 
three types of cells, i.e., R, G, and B cells have 
nearly the same spectral sensitivities to the light 

20 source wavelength, one pixel of the reflected light 

image can be formed using these four pixels. For 
example, Gl and Rl cells store non-emission-timing 
charges, and Bl and G3 cells store emission-timing 
charges. Then, the differences between Bl and Gl, and 

25 between G3 and Rl are added to each other, thus 

obtaining a reflected light image having the same 
resolution as that of the visible light image. 



Furthermore, if the difference between Bl and Gl or 
between G3 and Rl is considered as one pixel, the 
horizontal direction can be doubled with respect to the 
visible light image. 

Even when "G" cells only can be used to form a 
reflected light image, a reflected light image having 
the same resolution as that of the visible light image 
can be obtained. If a filter matrix shown in FIG. 10 
is used, and if Gl stores a non-emission-timing charge 
and G2 stores an emission-timing charge, their 
difference can be output. Since the difference circuit 
is normally designed to calculate the difference 
between pixels in an identical line, when the filter 
matrix shown in FIG. 9 is used, it is difficult to 
output the difference between Gl and G3 , but it is easy 
to output the difference between Gl and G2 in FIG. 10. 

Since the above-mentioned control for obtaining a 
reflected light image is different from that for 
obtaining a visible light image, the sensor control 
must be switched. The control may be explicitly 
switched by a hardware switch or a software switch 
(when this apparatus is connected to a personal 
computer and is used) . Alternatively, a mechanism for 
turning on/off the switch upon sliding the filter may 
be provided, and the current position of the filter may 
be automatically recognized to switch the control. 

In case of the arrangement shown in FIG. 8, it is 



easy to separately obtain a reflected light image and 
visible light image but it is difficult to simultane- 
ously obtain them. However, these images can be 
simultaneously obtained when a mechanism for moving the 
optical filter at high speed is added. Also, a method 
of rotating a circular filter having semi-circular 
filter elements is available. 

FIG. 11 shows another example of the arrangement 
that can simultaneously sense a reflected light image 
and visible light image using a single sensor. Using a 
light ray splitting means 34 such as a dichroic mirror 
or the like, which is similar to that used in FIG. 7, 
light is split into object light of the light source 
and other light components. Shutter means 35 and 36 
which can selectively pass or intercept these light 
components are provided. The light components that 
have passed through the shutter means are synthesized, 
and the synthesized light is guided to an optical 
sensor 38. 

The shutter means can use liquid crystal panels or 
the like. Since liquid crystal panels or the like can 
be non-mechanically driven, reliability can be improved 
as compared to a mechanical structure. 

In FIG, 11, object reflected light 39 of the light 
source is intercepted, and other light components 40 
can pass through the shutter. In the other state, the 
object reflected light reaches the sensor, but visible 
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light is intercepted by the shutter. 

In FIG. 11, two split light beams are synthesized 
again via the corresponding shutter means. If an 
element, the filter characteristics of which can be 
5 dynamically changed, is used, the optical path need not 

be split. In an arrangement shown in FIG. 12, an 
element 41 which can be switched between two states: 

(1) only the light source wavelength is passed; 

and 

10 (2) only visible light is passed 

is inserted between the len^ and sensor. By only 
switching the element state, two different types of 
images can be sensed. In FIG. 12, object reflected 
light 39 of the light source is intercepted, and other 

15 light components 40 can pass through the element. In 

the other state, the object reflected light reaches the 
sensor, but visible light is intercepted by the shutter. 

If an element which can be switched between two 
states : 

20 (1) only the light source wavelength is passed; 

and 

(2) all light components are passed 

and an element which can be switched between two 
states : 

25 (1) only visible light is passed; and 

(2) all light components are passed 
are available, the same arrangement can be realized by 



superposing these elements. 

When the color filter of the sensor has a special 
arrangement, both a reflected light image and visible 
light image can be obtained by a single sensor. 
FIG. 10 shows an example of the normal color filter 
matrix. However, when a matrix shown in FIG. 13 is 
used, cells can be classified into those for a 
reflected light image, and those for a visible light 
image. In FIG. 13, cells indicated by "H" have filters 
for passing only the wavelength of the light source. R 
G, and B cells are used for sensing a visible light 
image, and H cells are used for sensing a reflected 
light image. 

For example, HI and H3 cells store non- 
emission-timing charges, and H2 and H4 cells store 
emission-timing charges. When HI to H4 cells form one 
pixel of the reflected light image, the obtained 
reflected light image can have the same resolution as 
that of the visible light image. In a normal imaging 
camera, a filter for cutting infrared light components 
is used. When near infrared light is used as the light 
source, the light source emits light with higher 
intensity or infrared cut filters are attached onto 
only cells for the visible light image. 

FIG. 14 shows another example of the filter matrix 
As compared to FIG. 13, the vertical resolution is 
doubled, and the horizontal resolution is reduced 



to 2/3. If an image with higher resolution is required, 
this matrix is preferably used. 

The present invention described above can be 
improved by software processing using a versatile 
processor without using any dedicated hardware. For 
example, the processing shown in FIG. 4 can be 
implemented using a computer program, and when the 
computer program is installed in and executed by a 
computer via a recording medium such as a floppy disk, 
CD-ROM, or the like, the present invention can be 
practiced. 

Additional advantages and modifications will 
readily occur to those skilled in the art. Therefore, 
the invention in its broader aspects is not limited to 
the specific details and representative embodiments 
shown and described herein. Accordingly, various 
modifications may be made without departing from the 
spirit or scope of the general inventive concept as 
defined by the appended claims and their equivalents. 



